<!DOCTYPE html>
<html id="docs" lang="en" class="">
	<head>
	<meta charset="utf-8">
<title>Troubleshooting - Kubernetes</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="shortcut icon" type="image/png" href="../../../../images/favicon.png">
<link rel="stylesheet" type="text/css" href="../../../../css/base_fonts.css">
<link rel="stylesheet" type="text/css" href="../../../../css/styles.css">
<link rel="stylesheet" type="text/css" href="https://code.jquery.com/ui/1.12.1/themes/smoothness/jquery-ui.css">
<link rel="stylesheet" type="text/css" href="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.css">
<link rel="stylesheet" type="text/css" href="../../../../css/callouts.css">
<link rel="stylesheet" type="text/css" href="../../../../css/custom-jekyll/tags.css">




<meta name="description" content="Troubleshooting" />
<meta property="og:description" content="Troubleshooting" />

<meta property="og:url" content="https://kubernetes.io/docs/getting-started-guides/ubuntu/troubleshooting/" />
<meta property="og:title" content="Troubleshooting - Kubernetes" />

<script
src="https://code.jquery.com/jquery-3.2.1.min.js"
integrity="sha256-hwg4gsxgFZhOsEEamdOYGBf13FyQuiTwlAQgxVSNgt4="
crossorigin="anonymous"></script>
<script
src="https://code.jquery.com/ui/1.12.1/jquery-ui.min.js"
integrity="sha256-VazP97ZCwtekAsvgPBSUwPFKdrwD3unUfSGVYrahUqU="
crossorigin="anonymous"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/sweetalert/1.1.3/sweetalert.min.js"></script>
<script src="../../../../js/script.js"></script>
<script src="../../../../js/custom-jekyll/tags.js"></script>


	</head>
	<body>
		<div id="cellophane" onclick="kub.toggleMenu()"></div>

<header>
    <a href="../../../../index.html" class="logo"></a>

    <div class="nav-buttons" data-auto-burger="primary">
        <ul class="global-nav">
            
            
            <li><a href="../../../home.1">Documentation</a></li>
            
            <li><a href="../../../../blog/index.html">Blog</a></li>
            
            <li><a href="../../../../partners/index.html">Partners</a></li>
            
            <li><a href="../../../../community/index.html">Community</a></li>
            
            <li><a href="../../../../case-studies/index.html">Case Studies</a></li>
            
            
             <li>
                <a href="index.html#">
                    English <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="../../../../zh/index.html">中文 Chinese</a></li>
                
                    <li><a href="../../../../ko/index.html">한국어 Korean</a></li>
                
                </ul>
            </li>
         
            <li>
                <a href="index.html#">
                    v1.11 <span class="ui-icon ui-icon-carat-1-s"></span>
                </a>
                <ul>
                
                    <li><a href="https://kubernetes.io">v1.12</a></li>
                
                    <li><a href="../../../../index.html">v1.11</a></li>
                
                    <li><a href="https://v1-10.docs.kubernetes.io">v1.10</a></li>
                
                    <li><a href="https://v1-9.docs.kubernetes.io">v1.9</a></li>
                
                </ul>
            </li>
        </ul>
        
        <a href="../../../tutorials/kubernetes-basics/index.html" class="button" id="tryKubernetes" data-auto-burger-exclude>Try Kubernetes</a>
        <button id="hamburger" onclick="kub.toggleMenu()" data-auto-burger-exclude><div></div></button>
    </div>

    <nav id="mainNav">
        <main data-auto-burger="primary">
        <div class="nav-box">
            <h3><a href="../../../tutorials/stateless-application/hello-minikube/index.html">Get Started</a></h3>
            <p>Ready to get your hands dirty? Build a simple Kubernetes cluster that runs "Hello World" for Node.js.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../home.1">Documentation</a></h3>
            <p>Learn how to use Kubernetes with the use of walkthroughs, samples, and reference documentation. You can even <a href="../../../../editdocs/index.html" data-auto-burger-exclude>help contribute to the docs</a>!</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../../community/index.html">Community</a></h3>
            <p>If you need help, you can connect with other Kubernetes users and the Kubernetes authors, attend community events, and watch video presentations from around the web.</p>
        </div>
        <div class="nav-box">
            <h3><a href="../../../../blog/index.html">Blog</a></h3>
            <p>Read the latest news for Kubernetes and the containers space in general, and get technical how-tos hot off the presses.</p>
        </div>
        </main>
        <main data-auto-burger="primary">
        <div class="left">
            <h5 class="github-invite">Interested in hacking on the core Kubernetes code base?</h5>
            <a href="https://github.com/kubernetes/kubernetes" class="button" data-auto-burger-exclude>View On Github</a>
        </div>

        <div class="right">
            <h5 class="github-invite">Explore the community</h5>
            <div class="social">
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>Twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
        </div>
        <div class="clear" style="clear: both"></div>
        </main>
    </nav>
</header>

		
		
		<section id="hero" class="light-text no-sub">
			
















<div id="vendorStrip" class="light-text">
	<ul>
		
		
		<li><a href="../../../home.1">DOCUMENTATION</a></li>
		
		
		<li><a href="../../../setup/index.html">SETUP</a></li>
		
		
		<li><a href="../../../concepts/index.html">CONCEPTS</a></li>
		
		
		<li><a href="../../../tasks/index.html">TASKS</a></li>
		
		
		<li><a href="../../../tutorials/index.html">TUTORIALS</a></li>
		
		
		<li><a href="../../../reference.1">REFERENCE</a></li>
		
	</ul>
	<div id="searchBox">
		<input type="text" id="search" placeholder="Search" onkeydown="if (event.keyCode==13) window.location.replace('/docs/search/?q=' + this.value)" autofocus="autofocus">
	</div>
</div>

		</section>
		
		
<section id="deprecationWarning">
  <main>
    <div class="content deprecation-warning">
      <h3>
        Documentation for Kubernetes v1.11 is no longer actively maintained. The version you are currently viewing is a static snapshot.
        For up-to-date documentation, see the <a href="https://kubernetes.io/docs/home/">latest</a> version.
      </h3>
    </div>
  </main>
</section>


		<section id="encyclopedia">
			
<div id="docsToc">
     <div class="pi-accordion">
    	
        
        
        
        
        
         
             
                 
             
         
             
                 
             
         
             
                 
             
         
             
                 
                          
                          
                 
             
         
             
         
             
         
             
         
             
         
         
        
        <a class="item" data-title="Independent Solutions" href="../../index.html"></a>

	
	
		
		
	<div class="item" data-title="Bare Metal">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Fedora (Multi Node)" href="../../fedora/flannel_multi_node_cluster/index.html"></a>

		
	
		
		
<a class="item" data-title="Fedora (Single Node)" href="../../fedora/fedora_manual_config.1"></a>

		
	

		</div>
	</div>

		
	
		
		
	<div class="item" data-title="Kubernetes on Ubuntu">
		<div class="container">
		
		
	
	
		
		
<a class="item" data-title="Backups" href="../backups.1"></a>

		
	
		
		
<a class="item" data-title="Decommissioning" href="../decommissioning/index.html"></a>

		
	
		
		
<a class="item" data-title="Glossary and Terminology" href="../glossary/index.html"></a>

		
	
		
		
<a class="item" data-title="Local Kubernetes development with LXD" href="../local/index.html"></a>

		
	
		
		
<a class="item" data-title="Logging" href="../logging.1"></a>

		
	
		
		
<a class="item" data-title="Monitoring" href="../monitoring/index.html"></a>

		
	
		
		
<a class="item" data-title="Networking" href="../networking/index.html"></a>

		
	
		
		
<a class="item" data-title="Operational Considerations" href="../operational-considerations/index.html"></a>

		
	
		
		
<a class="item" data-title="Rancher Integration with Ubuntu Kubernetes" href="../rancher/index.html"></a>

		
	
		
		
<a class="item" data-title="Scaling" href="../scaling/index.html"></a>

		
	
		
		
<a class="item" data-title="Security Considerations" href="../security/index.html"></a>

		
	
		
		
<a class="item" data-title="Setting up Kubernetes with Juju" href="../installation/index.html"></a>

		
	
		
		
<a class="item" data-title="Storage" href="../storage/index.html"></a>

		
	
		
		
<a class="item" data-title="Troubleshooting" href="index.html"></a>

		
	
		
		
<a class="item" data-title="Upgrades" href="../upgrades/index.html"></a>

		
	
		
		
<a class="item" data-title="Validation - End-to-end Testing" href="../validation.1"></a>

		
	

		</div>
	</div>

		
	
		
		
<a class="item" data-title="Deprecated Alternatives" href="../../alternatives/index.html"></a>

		
	






     </div> 
    <button class="push-menu-close-button" onclick="kub.toggleToc()"></button>
</div> 

			<div id="docsContent">
				
<p><a href="../../../editdocs#docs/getting-started-guides/ubuntu/troubleshooting.md" id="editPageButton">Edit This Page</a></p>

<h1>Troubleshooting</h1>




<p>This document with highlighting how to troubleshoot the deployment of a Kubernetes cluster, it will not cover debugging of workloads inside Kubernetes.</p>











<ul id="markdown-toc">










<li><a href="index.html#before-you-begin">Before you begin</a></li>












<li><a href="index.html#understanding-cluster-status">Understanding Cluster Status</a></li>




<li><a href="index.html#sshing-to-units">SSHing to units</a></li>




<li><a href="index.html#collecting-debug-information">Collecting debug information</a></li>




<li><a href="index.html#common-problems">Common Problems</a></li>




<li><a href="index.html#logging-and-monitoring">Logging and monitoring</a></li>



























</ul>



<h2 id="before-you-begin">Before you begin</h2>
<p>This page assumes you have a working Juju deployed cluster.</p>




<h2 id="understanding-cluster-status">Understanding Cluster Status</h2>

<p>Using <code>juju status</code> can give you some insight as to what&rsquo;s happening in a cluster:</p>

<pre><code>Model  Controller  Cloud/Region   Version
kubes  work-multi  aws/us-east-2  2.0.2.1

App                Version  Status  Scale  Charm              Store       Rev  OS      Notes
easyrsa            3.0.1    active      1  easyrsa            jujucharms    3  ubuntu  
etcd               2.2.5    active      1  etcd               jujucharms   17  ubuntu  
flannel            0.6.1    active      2  flannel            jujucharms    6  ubuntu  
kubernetes-master  1.4.5    active      1  kubernetes-master  jujucharms    8  ubuntu  exposed
kubernetes-worker  1.4.5    active      1  kubernetes-worker  jujucharms   11  ubuntu  exposed

Unit                  Workload  Agent  Machine  Public address  Ports           Message
easyrsa/0*            active    idle   0/lxd/0  10.0.0.55                       Certificate Authority connected.
etcd/0*               active    idle   0        52.15.47.228    2379/tcp        Healthy with 1 known peers.
kubernetes-master/0*  active    idle   0        52.15.47.228    6443/tcp        Kubernetes master services ready.
  flannel/1           active    idle            52.15.47.228                    Flannel subnet 10.1.75.1/24
kubernetes-worker/0*  active    idle   1        52.15.177.233   80/tcp,443/tcp  Kubernetes worker running.
  flannel/0*          active    idle            52.15.177.233                   Flannel subnet 10.1.63.1/24

Machine  State    DNS            Inst id              Series  AZ
0        started  52.15.47.228   i-0bb211a18be691473  xenial  us-east-2a
0/lxd/0  started  10.0.0.55      juju-153b74-0-lxd-0  xenial  
1        started  52.15.177.233  i-0502d7de733be31bb  xenial  us-east-2b
</code></pre>

<p>In this example we can glean some information. The <code>Workload</code> column will show the status of a given service. The <code>Message</code> section will show you the health of a given service in the cluster. During deployment and maintenance these workload statuses will update to reflect what a given node is doing. For example the workload my say <code>maintenance</code> while message will describe this maintenance as <code>Installing docker</code>.</p>

<p>During normal operation the Workload should read <code>active</code>, the Agent column (which reflects what the Juju agent is doing) should read <code>idle</code>, and the messages will either say <code>Ready</code> or another descriptive term. <code>juju status --color</code> will also return all green results when a cluster&rsquo;s deployment is healthy.</p>

<p>Status can become unwieldy for large clusters, it is then recommended to check status on individual services, for example to check the status on the workers only:</p>

<pre><code>juju status kubernetes-worker
</code></pre>

<p>or just on the etcd cluster:</p>

<pre><code>juju status etcd
</code></pre>

<p>Errors will have an obvious message, and will return a red result when used with <code>juju status --color</code>. Nodes that come up in this manner should be investigated.</p>

<h2 id="sshing-to-units">SSHing to units</h2>

<p>You can ssh to individual units easily with the following convention, <code>juju ssh &lt;servicename&gt;/&lt;unit#&gt;</code>:</p>

<pre><code>juju ssh kubernetes-worker/3
</code></pre>

<p>Will automatically ssh you to the 3rd worker unit.</p>

<pre><code>juju ssh easyrsa/0 
</code></pre>

<p>This will automatically ssh you to the easyrsa unit.</p>

<h2 id="collecting-debug-information">Collecting debug information</h2>

<p>Sometimes it is useful to collect all the information from a cluster to share with a developer to identify problems. This is best accomplished with <a href="https://github.com/juju-solutions/cdk-field-agent" target="_blank">CDK Field Agent</a>.</p>

<p>Download and execute the collect.py script from <a href="https://github.com/juju-solutions/cdk-field-agent" target="_blank">CDK Field Agent</a> on a box that has a Juju client configured with the current controller and model pointing at the CDK deployment of interest.</p>

<p>Running the script will generate a tarball of system information and includes basic information such as systemctl status, Juju logs, charm unit data, etc. Additional application-specific information may be included as well.</p>

<h2 id="common-problems">Common Problems</h2>

<h3 id="load-balancer-interfering-with-helm">Load Balancer interfering with Helm</h3>

<p>This section assumes you have a working deployment of Kubernetes via Juju using a Load Balancer for the API, and that you are using Helm to deploy charts.</p>

<p>To deploy Helm you will have run:</p>

<pre><code>helm init
$HELM_HOME has been configured at /home/ubuntu/.helm
Tiller (the helm server side component) has been installed into your Kubernetes Cluster.
Happy Helming!
</code></pre>

<p>Then when using helm you may see one of the following errors:</p>

<ul>
<li>Helm doesn&rsquo;t get the version from the Tiller server</li>
</ul>

<pre><code>helm version
Client: &amp;version.Version{SemVer:&quot;v2.1.3&quot;, GitCommit:&quot;5cbc48fb305ca4bf68c26eb8d2a7eb363227e973&quot;, GitTreeState:&quot;clean&quot;}
Error: cannot connect to Tiller
</code></pre>

<ul>
<li>Helm cannot install your chart</li>
</ul>

<pre><code>helm install &lt;chart&gt; --debug
Error: forwarding ports: error upgrading connection: Upgrade request required
</code></pre>

<p>This is caused by the API load balancer not forwarding ports in the context of the helm client-server relationship. To deploy using helm, you will need to follow these steps:</p>

<ol>
<li>Expose the Kubernetes Master service</li>
</ol>

<pre><code>   juju expose kubernetes-master
</code></pre>

<ol>
<li>Identify the public IP address of one of your masters</li>
</ol>

<pre><code>   juju status kubernetes-master
   Model       Controller  Cloud/Region   Version
   production  k8s-admin   aws/us-east-1  2.0.0

   App                Version  Status  Scale  Charm              Store       Rev  OS      Notes
   flannel            0.6.1    active      1  flannel            jujucharms    7  ubuntu
   kubernetes-master  1.5.1    active      1  kubernetes-master  jujucharms   10  ubuntu  exposed

   Unit                  Workload  Agent  Machine  Public address  Ports     Message
   kubernetes-master/0*  active    idle   5        54.210.100.102    6443/tcp  Kubernetes master running.
     flannel/0           active    idle            54.210.100.102              Flannel subnet 10.1.50.1/24

   Machine  State    DNS           Inst id              Series  AZ
   5        started  54.210.100.102  i-002b7150639eb183b  xenial  us-east-1a

   Relation      Provides               Consumes           Type
   certificates  easyrsa                kubernetes-master  regular
   etcd          etcd                   flannel            regular
   etcd          etcd                   kubernetes-master  regular
   cni           flannel                kubernetes-master  regular
   loadbalancer  kubeapi-load-balancer  kubernetes-master  regular
   cni           kubernetes-master      flannel            subordinate
   cluster-dns   kubernetes-master      kubernetes-worker  regular
   cni           kubernetes-worker      flannel            subordinate
</code></pre>

<p>In this context the public IP address is 54.210.100.102.</p>

<p>If you want to access this data programmatically you can use the JSON output:</p>

<pre><code>   juju show-status kubernetes-master --format json | jq --raw-output '.applications.&quot;kubernetes-master&quot;.units | keys[]'
   54.210.100.102
</code></pre>

<ol>
<li>Update the kubeconfig file</li>
</ol>

<p>Identify the kubeconfig file or section used for this cluster, and edit the server configuration.</p>

<p>By default, it will look like <code>https://54.213.123.123:443</code>. Replace it with the Kubernetes Master endpoint <code>https://54.210.100.102:6443</code> and save.</p>

<p>Note that the default port used by CDK for the Kubernetes Master API is 6443 while the port exposed by the load balancer is 443.</p>

<ol>
<li>Start helm again!</li>
</ol>

<pre><code>   helm install &lt;chart&gt; --debug
   Created tunnel using local port: '36749'
   SERVER: &quot;localhost:36749&quot;
   CHART PATH: /home/ubuntu/.helm/&lt;chart&gt;
   NAME:   &lt;chart&gt;
   ...
   ...
</code></pre>

<h2 id="logging-and-monitoring">Logging and monitoring</h2>

<p>By default there is no log aggregation of the Kubernetes nodes, each node logs locally. Please read over the <a href="https://kubernetes.io/docs/getting-started-guides/ubuntu/logging/" target="_blank">logging</a> page for more information.</p>






<p>
</p>










				<div class="issue-button-container">
					<p><a href="index.html"><img src="https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/ubuntu/troubleshooting.md?pixel" alt="Analytics" /></a></p>
					
					
					<script type="text/javascript">
					PDRTJS_settings_8345992 = {
					"id" : "8345992",
					"unique_id" : "\/docs\/getting-started-guides\/ubuntu\/troubleshooting\/",
					"title" : "Troubleshooting",
					"permalink" : "https:\/\/kubernetes.io\/docs\/getting-started-guides\/ubuntu\/troubleshooting\/"
					};
					(function(d,c,j){if(!document.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src=('https:'==document.location.protocol)?'https://polldaddy.com/js/rating/rating.js':'http://i0.poll.fm/js/rating/rating.js';s=document.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);}}(document,'script','pd-rating-js'));
					</script>
					<a href="index.html" onclick="window.open('https://github.com/kubernetes/website/issues/new?title=Issue%20with%20' +
					'k8s.io'+window.location.pathname)" class="button issue">Create an Issue</a>
					
					
					
					<a href="../../../editdocs#docs/getting-started-guides/ubuntu/troubleshooting.md" class="button issue">Edit this Page</a>
					
				</div>
			</div>
		</section>
		<footer>
    <main class="light-text">
        <nav>
            
            
            
            <a href="../../../home.1">Documentation</a>
            
            <a href="../../../../blog/index.html">Blog</a>
            
            <a href="../../../../partners/index.html">Partners</a>
            
            <a href="../../../../community/index.html">Community</a>
            
            <a href="../../../../case-studies/index.html">Case Studies</a>
            
        </nav>
        <div class="social">
            <div>
                <a href="https://twitter.com/kubernetesio" class="twitter"><span>twitter</span></a>
                <a href="https://github.com/kubernetes/kubernetes" class="github"><span>Github</span></a>
                <a href="http://slack.k8s.io/" class="slack"><span>Slack</span></a>
            </div>
            <div>
                <a href="http://stackoverflow.com/questions/tagged/kubernetes" class="stack-overflow"><span>Stack Overflow</span></a>
                <a href="https://discuss.kubernetes.io" class="mailing-list"><span>Forum</span></a>
                <a href="https://calendar.google.com/calendar/embed?src=nt2tcnbtbied3l6gi2h29slvc0%40group.calendar.google.com" class="calendar"><span>Events Calendar</span></a>
            </div>
            <div>
                <a href="../../index.html" class="button">Get Kubernetes</a>
                <a href="https://git.k8s.io/community/contributors/guide" class="button">Contribute</a>
            </div>
        </div>
        <div id="miceType" class="center">
            &copy; 2018 The Kubernetes Authors | Documentation Distributed under <a href="https://git.k8s.io/website/LICENSE" class="light-text">CC BY 4.0</a>
        </div>
        <div id="miceType" class="center">
            Copyright &copy; 2018 The Linux Foundation&reg;. All rights reserved. The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our <a href="https://www.linuxfoundation.org/trademark-usage" class="light-text">Trademark Usage page</a>
        </div>
    </main>
</footer>

		<button class="flyout-button" onclick="kub.toggleToc()"></button>

<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
    (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-36037335-10', 'auto');
ga('send', 'pageview');


(function () {
    window.addEventListener('DOMContentLoaded', init)

        
        function init() {
            window.removeEventListener('DOMContentLoaded', init)
                hideNav()
        }

    function hideNav(toc){
        if (!toc) toc = document.querySelector('#docsToc')
        if (!toc) return
            var container = toc.querySelector('.container')

                
                if (container) {
                    if (container.childElementCount === 0 || toc.querySelectorAll('a.item').length === 1) {
                        toc.style.display = 'none'
                            document.getElementById('docsContent').style.width = '100%'
                    }
                } else {
                    requestAnimationFrame(function () {
                        hideNav(toc)
                    })
                }
    }
})();
</script>



	</body>
</html>